<html>
<head><meta charset="utf-8"><title>is there a way to replicate CI locally? · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html">is there a way to replicate CI locally?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="228184807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228184807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228184807">(Mar 01 2021 at 05:46)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/pull/81601">https://github.com/rust-lang/rust/pull/81601</a> is consistently failing because of an environment that's different than what I have locally. Is there some way to replicate it, maybe with a docker image? It only goes wrong after the PR's already been approved, so it's very hard to debug. I've tried running <code>src/ci/init_repo.sh</code> but it gives an error that <code>rustsrc</code> is missing.</p>



<a name="228184929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228184929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228184929">(Mar 01 2021 at 05:49)</a>:</h4>
<p>I think the root cause is that CI is playing games with submodules to try and pretend that the git history exists when it doesn't. Why does it do that? It would make things so much easier if I could just assume that <code>.git</code> means it has git history available.</p>



<a name="228197826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228197826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228197826">(Mar 01 2021 at 08:55)</a>:</h4>
<p>I think the commit of several submodules ends up in the dist packages.</p>



<a name="228202577"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228202577" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228202577">(Mar 01 2021 at 09:42)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> it's doing that to speed up CI, since cloning llvm-project is slow</p>



<a name="228230795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228230795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228230795">(Mar 01 2021 at 13:40)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> how can I replicate it locally? Where is it downloading things from?</p>



<a name="228230864"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228230864" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228230864">(Mar 01 2021 at 13:41)</a>:</h4>
<p>you should try running the script on a fresh clone of rustc, since it messes stuff up</p>



<a name="228230963"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228230963" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228230963">(Mar 01 2021 at 13:41)</a>:</h4>
<p>it downloads the <code>.tar.gz</code> tarball of the latest commit from github and extracts it</p>



<a name="228231028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228231028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228231028">(Mar 01 2021 at 13:42)</a>:</h4>
<p>instead of doing a git clone</p>



<a name="228232838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228232838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228232838">(Mar 01 2021 at 13:53)</a>:</h4>
<p>is there a way to see exactly what CI is doing without having to search through the logs? Like a script that github generates or something?</p>



<a name="228233136"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228233136" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228233136">(Mar 01 2021 at 13:55)</a>:</h4>
<p>I tried looking for <code>[command]</code> but apparently that <em>only</em> shows git invocations, not bootstrap or <code>./configure</code></p>



<a name="228233493"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228233493" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228233493">(Mar 01 2021 at 13:57)</a>:</h4>
<p>(so I don't forget: <code>'\[command\]' ci.log  | cut -d ' ' -f 3- | sed 's/^/git /'</code> shows those)</p>



<a name="228233868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228233868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228233868">(Mar 01 2021 at 14:00)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> not really</p>



<a name="228233906"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228233906" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228233906">(Mar 01 2021 at 14:00)</a>:</h4>
<p>just running that script should reproduce tho</p>



<a name="228234289"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228234289" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228234289">(Mar 01 2021 at 14:02)</a>:</h4>
<p>ok I found the issue, <code>init_repo</code> can't be run by itself, it has to be run through <code>checkout-submodules.sh</code></p>



<a name="228235523"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228235523" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228235523">(Mar 01 2021 at 14:10)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span>  it fails before it gets to my change:</p>
<div class="codehilite"><pre><span></span><code>Dist rustc-1.52.0-dev-src
Error: failed to generate tarballs

Caused by:
    0: failed to create file &#39;/home/joshua/src/rust/rust/build/dist/rustc-1.52.0-dev-src.tar.gz&#39;
    1: No such file or directory (os error 2)


command did not execute successfully: &quot;/home/joshua/src/rust/rust/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate&quot; &quot;tarball&quot; &quot;--input&quot; &quot;/home/joshua/src/rust/rust/build/tmp/tarball/rustc/src/rustc-1.52.0-dev-src&quot; &quot;--output&quot; &quot;/home/joshua/src/rust/rust/build/dist/rustc-1.52.0-dev-src&quot; &quot;--work-dir&quot; &quot;/home/joshua/src/rust/rust/build/tmp/tarball/rustc/src&quot;
expected success, got: exit code: 1
</code></pre></div>



<a name="228235596"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228235596" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228235596">(Mar 01 2021 at 14:10)</a>:</h4>
<p><code>build/dist</code> doesn't seem to exist</p>



<a name="228235617"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228235617" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228235617">(Mar 01 2021 at 14:10)</a>:</h4>
<p>I'll just make it manually for now</p>



<a name="228235664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228235664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228235664">(Mar 01 2021 at 14:10)</a>:</h4>
<p>huh that's weird</p>



<a name="228237194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228237194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228237194">(Mar 01 2021 at 14:19)</a>:</h4>
<p>ok yay I managed to replicate locally <span aria-label="tada" class="emoji emoji-1f389" role="img" title="tada">:tada:</span></p>



<a name="228237378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228237378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228237378">(Mar 01 2021 at 14:20)</a>:</h4>
<p>so I don't forget, the steps were the git things above, plus</p>
<div class="codehilite"><pre><span></span><code>src/ci/scripts/checkout-submodules.sh
./configure --build=x86_64-unknown-linux-gnu --set rust.ignore-git=false
mkdir build/dist
python3 ./x.py --stage 2 test distcheck
</code></pre></div>



<a name="228239177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228239177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228239177">(Mar 01 2021 at 14:29)</a>:</h4>
<p>I've made more progress: the error occurs because <code>distcheck</code> makes its own copy of the source tree in <code>build/tmp/distcheck</code> (??) and the git state there is different than in the main tree</p>



<a name="228239711"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228239711" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228239711">(Mar 01 2021 at 14:32)</a>:</h4>
<p>backtrace:</p>
<div class="codehilite"><pre><span></span><code>Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -&gt; x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
thread &#39;main&#39; panicked at &#39;called `Option::unwrap()` on a `None` value&#39;, src/bootstrap/native.rs:123:43
stack backtrace:
   0: rust_begin_unwind
             at /rustc/14b27555a03f4abf9b8d80ea34f59144e2ded5e4/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/14b27555a03f4abf9b8d80ea34f59144e2ded5e4/library/core/src/panicking.rs:92:14
   2: core::panicking::panic
             at /rustc/14b27555a03f4abf9b8d80ea34f59144e2ded5e4/library/core/src/panicking.rs:50:5
   3: core::option::Option&lt;T&gt;::unwrap
             at /rustc/14b27555a03f4abf9b8d80ea34f59144e2ded5e4/library/core/src/option.rs:386:21
   4: bootstrap::native::update_llvm_submodule
             at ./src/bootstrap/native.rs:123:16
   5: &lt;bootstrap::native::Llvm as bootstrap::builder::Step&gt;::run
             at ./src/bootstrap/native.rs:206:13
   6: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1522:23
   7: bootstrap::compile::rustc_cargo_env
             at ./src/bootstrap/compile.rs:644:27
   8: bootstrap::compile::rustc_cargo
             at ./src/bootstrap/compile.rs:596:5
   9: &lt;bootstrap::compile::Rustc as bootstrap::builder::Step&gt;::run
             at ./src/bootstrap/compile.rs:532:9
  10: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1522:23
  11: &lt;bootstrap::compile::Assemble as bootstrap::builder::Step&gt;::run
             at ./src/bootstrap/compile.rs:1014:9
  12: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1522:23
  13: bootstrap::builder::Builder::compiler
             at ./src/bootstrap/builder.rs:596:9
  14: &lt;bootstrap::compile::Assemble as bootstrap::builder::Step&gt;::run
             at ./src/bootstrap/compile.rs:1001:30
  15: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1522:23
  16: bootstrap::builder::Builder::compiler
             at ./src/bootstrap/builder.rs:596:9
  17: &lt;bootstrap::test::Ui as bootstrap::builder::Step&gt;::make_run
             at ./src/bootstrap/test.rs:918:32
  18: bootstrap::builder::StepDescription::maybe_run
             at ./src/bootstrap/builder.rs:197:13
  19: bootstrap::builder::StepDescription::run
             at ./src/bootstrap/builder.rs:218:25
  20: bootstrap::builder::Builder::run_step_descriptions
             at ./src/bootstrap/builder.rs:588:9
  21: bootstrap::builder::Builder::execute_cli
             at ./src/bootstrap/builder.rs:580:9
  22: bootstrap::Build::build
             at ./src/bootstrap/lib.rs:510:13
  23: bootstrap::main
             at ./src/bootstrap/bin/main.rs:33:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/14b27555a03f4abf9b8d80ea34f59144e2ded5e4/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
failed to run: /home/joshua/src/rust/rust/build/tmp/distcheck/build/bootstrap/debug/bootstrap test --stage 2
</code></pre></div>



<a name="228240089"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228240089" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228240089">(Mar 01 2021 at 14:34)</a>:</h4>
<p>ok cool, the issue is that the source tree has no <code>.git</code> directory. I can add that check and just skip submodules if so</p>



<a name="228242586"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/is%20there%20a%20way%20to%20replicate%20CI%20locally%3F/near/228242586" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/is.20there.20a.20way.20to.20replicate.20CI.20locally.3F.html#228242586">(Mar 01 2021 at 14:48)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/pull/81601/commits/6bda460cb0d21acef3cf44cc2ea8b379abc7d5c4">https://github.com/rust-lang/rust/pull/81601/commits/6bda460cb0d21acef3cf44cc2ea8b379abc7d5c4</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>